{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## 1. Introduction of PP-ShiTu\n",
    "PP-ShiTu is a practical lightweight general-purpose image recognition system, which mainly consists of three modules: subject detection, feature learning and vector retrieval. The system adopts a variety of strategies from 8 aspects of backbone network selection and adjustment, loss function selection, data enhancement, learning rate transformation strategy, regularization parameter selection, use of pre-trained models, and model pruning and quantification Optimization, and finally a system that can complete the image recognition of the 10w+ library in only 0.2s on the CPU.\n",
    "For more details, please refer to [PP-ShiTu Technical Paper](https://arxiv.org/pdf/2111.00775.pdf).\n",
    "\n",
    "Learn more about PaddleClas at https://github.com/PaddlePaddle/PaddleClas.\n",
    "\n",
    "## 2. Preview and application scenarios\n",
    "### 2.1 product recognition：\n",
    "\n",
    "#### 2.1.1 dataset：\n",
    "\n",
    "The training data set and test set of PP-ShiTu are composed of 7 data sets such as Aliproduct and GLDv2. For details, please refer to [PP-ShiTu Experimental Part](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/docs/zh_CN/image_recognition_pipeline/feature_extraction.md#4-%E5%AE%9E%E9%AA%8C%E9%83%A8%E5%88%86)\n",
    "\n",
    "#### 2.1.2 output preview：\n",
    "\n",
    "The detection effect of PP-ShiTu on the picture is as follows\n",
    "\n",
    "![](https://github.com/PaddlePaddle/PaddleClas/raw/release/2.4/docs/images/recognition/drink_data_demo/output/nongfu_spring.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. How to use\n",
    "\n",
    "### 3.1 model inference：\n",
    "\n",
    "- download PaddleClas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "execution": {
     "iopub.execute_input": "2022-11-08T08:24:16.514016Z",
     "iopub.status.busy": "2022-11-08T08:24:16.513368Z",
     "iopub.status.idle": "2022-11-08T08:25:00.630629Z",
     "shell.execute_reply": "2022-11-08T08:25:00.629113Z",
     "shell.execute_reply.started": "2022-11-08T08:24:16.513971Z"
    },
    "jupyter": {
     "outputs_hidden": true
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# When not running on the Jupyter Notebook, you need to add the \"!\" and \"%\" statement comments, do not need to run.\n",
    "%cd ~/work\n",
    "\n",
    "# Clone PaddleClas（Cloning is faster on gitee）\n",
    "!git clone https://gitee.com/paddlepaddle/PaddleClas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Install PaddleClas and its dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "execution": {
     "iopub.execute_input": "2022-11-08T08:26:02.622321Z",
     "iopub.status.busy": "2022-11-08T08:26:02.621656Z",
     "iopub.status.idle": "2022-11-08T08:26:05.016413Z",
     "shell.execute_reply": "2022-11-08T08:26:05.015052Z",
     "shell.execute_reply.started": "2022-11-08T08:26:02.622277Z"
    },
    "jupyter": {
     "outputs_hidden": true
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Access into PaddleClas\n",
    "%cd ~/work/PaddleClas/\n",
    "\n",
    "# Switch to the 2.4 branch\n",
    "!git checkout release/2.4\n",
    "\n",
    "# Install required dependencies\n",
    "!pip install -r requirements.txt\n",
    "\n",
    "# set GPU environment\n",
    "# %env CUDA_VISIBLE_DEVICES=0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Quick start\n",
    "\n",
    "Congratulations! You have successfully installed PaddleClas, now you can experience the image recognition as guided below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-08T08:26:11.091828Z",
     "iopub.status.busy": "2022-11-08T08:26:11.090376Z",
     "iopub.status.idle": "2022-11-08T08:29:06.202735Z",
     "shell.execute_reply": "2022-11-08T08:29:06.201197Z",
     "shell.execute_reply.started": "2022-11-08T08:26:11.091754Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Access into PaddleClas\n",
    "%cd ~/work/PaddleClas/\n",
    "\n",
    "# Create a folder for storing mainbody detection and feature extraction inference models\n",
    "%mkdir -p deploy/models\n",
    "\n",
    "# Access into models\n",
    "%cd deploy/models\n",
    "\n",
    "# Download the mainbody detection inference model and unzip it\n",
    "!wget -nc https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/models/inference/picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer.tar && tar -xf picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer.tar\n",
    "\n",
    "# Download the feature extraction inference model and unzip it\n",
    "!wget -nc https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/models/inference/general_PPLCNet_x2_5_lite_v1.0_infer.tar && tar -xf general_PPLCNet_x2_5_lite_v1.0_infer.tar\n",
    "\n",
    "# Back to deloy\n",
    "%cd ~/work/PaddleClas/deploy/\n",
    "\n",
    "# Download the test data drink_dataset_v1.0 and unzip it\n",
    "!wget -nc https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/data/drink_dataset_v1.0.tar && tar -xf drink_dataset_v1.0.tar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-08T08:31:30.364422Z",
     "iopub.status.busy": "2022-11-08T08:31:30.363351Z",
     "iopub.status.idle": "2022-11-08T08:31:37.682006Z",
     "shell.execute_reply": "2022-11-08T08:31:37.680563Z",
     "shell.execute_reply.started": "2022-11-08T08:31:30.364378Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Access into PaddleClas\n",
    "%cd ~/work/PaddleClas/\n",
    "\n",
    "# Access into deploy\n",
    "%cd ./deploy\n",
    "\n",
    "# general_PPLCNet_x2_5_lite_v1.0 inference model is used to extract the features of gallery images and make a retrieval database\n",
    "!python3.7 python/build_gallery.py -c configs/inference_general.yaml -o Global.rec_inference_model_dir=./models/general_PPLCNet_x2_5_lite_v1.0_infer\n",
    "\n",
    "# Recognition inference for nongfu_spring.jpeg image (GPU inference)\n",
    "!python3.7 python/predict_system.py -c configs/inference_general.yaml\n",
    "# Recognition inference for nongfu_spring.jpeg image (CPU inference)\n",
    "!python3.7 python/predict_system.py -c configs/inference_general.yaml -o Global.use_gpu=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the same time, the recognition results (with detection bounding box, predicted class name and similarity) will be saved to `PaddleClas/deploy/output/nongfu_spring.jpeg`, as [2.1.2 output preview](#212-output-preview) displayed.\n",
    "\n",
    "![](https://github.com/PaddlePaddle/PaddleClas/raw/release/2.4/docs/images/recognition/drink_data_demo/output/nongfu_spring.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Model training\n",
    "\n",
    "- clone PaddleClas repo(refer to [3.1 model inference](#31-model-inference)), and checkout to release/2.4 branch\n",
    "- For the dataset preparation, training, evaluation and other steps of the mainbody detection model, please refer to  [PP-ShiTu mainbody detection doc](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/docs/zh_CN/image_recognition_pipeline/mainbody_detection.md)\n",
    "- For the dataset preparation, training, evaluation and other steps of the mainbody detection model, please refer to  [PP-ShiTu feature extraction doc](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/docs/zh_CN/image_recognition_pipeline/feature_extraction.md#51-%E6%95%B0%E6%8D%AE%E5%87%86%E5%A4%87)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Algorithm\n",
    "PP-ShiTu series recognition systems, including PP-ShiTu introduced in this document, consists of three modules to complete the entire recognition process, as shown in the figure below\n",
    "\n",
    "![PP-ShiTu System](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.4/docs/images/structure.jpg?raw=true)\n",
    "\n",
    "- Mainbody detection: The blue colored module in the figure above detects potential targets in the input image, and then cropping these targets, filtering unimportant backgrounds, and reducing background interference. In fact, this practice of retaining the mainbody and filtering the background is a simple, effective and widely used method in practice.\n",
    "- Feature extraction: Receive the cropped image containing the target mainbody output by the **mainbody detection** module, and input it into the feature extraction model to obtain the corresponding feature vector, which is used as the representation feature of the image for subsequent retrieval.\n",
    "- Vector retrieval: Receive one or more feature vectors output by the **feature extraction** module, and retrieve them one by one in the vector library, finally return the retrieval result. This module does not require additional training and can be used by installing the third-party open source faiss retrieval library\n",
    "\n",
    "In the recognition system, one of the most important modules is the feature extraction model. The generalization of feature extraction model directly affects the quality of the vectors in the retrieval library and the vectors to be retrieved. Therefore, we will introduce feature extraction model below in 5 parts.\n",
    "\n",
    "- Backbone\n",
    "    The Backbone uses PP_LCNet_x2_5, which explores many effective structural design schemes for the performance optimization of Intel CPU. Finally, the backbone further improves the performance of the model without increasing the reasoning time, and finally greatly exceeds the existing SOTA model.\n",
    "\n",
    "- Neck\n",
    "\n",
    "    The Neck part uses FC Layer to reduce the dimension of features extracted by the Backbone, thus reducing the cost and computation of feature storage.\n",
    "\n",
    "- Head\n",
    "\n",
    "    The Head part uses ArcMargin. During training, margin was specified to increase the Angle difference between similar features before classification, so as to further improve the characterization ability of feature extraction.\n",
    "\n",
    "- Loss\n",
    "\n",
    "    The Loss adopts Cross entropy loss, using classification loss to optimize the network during training.See the General Identification Profile for detailed configuration files.\n",
    "\n",
    "## 5. Note\n",
    "PP-ShiTu is looking for the most cost-effective image recognition solution in industrial practice. However, considering that the datasets of different recognition scenarios have their own distribution characteristics, as well as the limitations of software and hardware during training, it is difficult to integrate all datasets at one time. Therefore, it is recommended that users, after understanding the characteristics of your actual datasets, fine-tune or even make an further development on your own datasets based on the PP-ShiTu pre-training model and training configuration, in order to obtain better performance and generalization.\n",
    "\n",
    "## 6. Reference\n",
    "```log\n",
    "@article{cui2021pp,\n",
    "    title={PP-LCNet: A Lightweight CPU Convolutional Neural Network},\n",
    "    author={Cui, Cheng and Gao, Tingquan and Wei, Shengyu and Du, Yuning and Guo, Ruoyu and Dong, Shuilong and Lu, Bin and Zhou, Ying and Lv, Xueying and Liu, Qiwen and others},\n",
    "    journal={arXiv preprint arXiv:2109.15099},\n",
    "    year={2021}\n",
    "}\n",
    "\n",
    "@article{wei2021pp,\n",
    "    title={PP-ShiTu: A Practical Lightweight Image Recognition System},\n",
    "    author={Wei, Shengyu and Guo, Ruoyu and Cui, Cheng and Lu, Bin and Dong, Shuilong and Gao, Tingquan and Du, Yuning and Zhou, Ying and Lyu, Xueying and Liu, Qiwen and others},\n",
    "    journal={arXiv preprint arXiv:2111.00775},\n",
    "    year={2021}\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "请点击[此处](https://ai.baidu.com/docs#/AIStudio_Project_Notebook/a38e5576)查看本环境基本用法.  <br>\n",
    "Please click [here ](https://ai.baidu.com/docs#/AIStudio_Project_Notebook/a38e5576) for more detailed instructions. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
